Evolving Data Archives

ABSTRACT

A data archive constructed from source data, whose structure and associated schemas can evolve based on the generated responses to user data requests. Based on the analysis of the responses, the schema and/or archive structure can be modified to provide greater knowledge, definition and operations available to be performed on the data, as well as to reduce the processing and storage costs associated with housing and accessing the data within the archive.

FIELD OF THE INVENTION

The field of the invention is data access and storage technologies.

BACKGROUND

The background description includes information that may be useful inunderstanding the present invention. It is not an admission that any ofthe information provided herein is prior art or relevant to thepresently claimed invention, or that any publication specifically orimplicitly referenced is prior art.

The evolution of computing and networking technologies have made datacollection, storage, and analysis increasingly easier to perform, and ata continuously larger scale. The ever-decreasing size of network-capablecomputing devices have increased the number of data sources gathering orcreating data, the types of data available from these sources, and theoverall amount of data available. Likewise, advancements in datacommunications and storage technologies have enabled entities interestedin data from these sources to collect increasingly larger amounts ofdata in databases or other data stores. This exponential growth indigitized data generation and collection continues to be fueled bymachine generated data in provenance of devices such as sensors andprobes that can monitor, measure and assert health, behavior, state,environment and performance of many types of machines and man-madesystems, as well as humans and many aspects of the natural world.

The collection of data on such a scale allows for analysis that canresult in discoveries and advancements, across various fields of study,which were not previously possible. For example, a medical researchercan use medical information gathered by wearable devices or sensorsoutside of a hospital setting to analyze health or medical patternsacross a population. In another example, an advertiser can use onlinebehaviors of a population of users to determine product trends,interests and advertisement effectiveness within a population.

However, certain data generating devices such as sensors or probes arecapable of generating data flows that, while digital, reflect theiranalog nature (and moreover, is often non-linear) or simply cannot beclassified as symbolic and human-readable.

To explore, discover and extract pertinent information out of these newdata flows, an incremental process is required that allows for startingat a state where very little is known about the data, and provides fordevelopment towards a full data model at both the data consumption sideand also at the data repository level. The complexity of this taskrequires methods far beyond simple numeric comparison and/or textualsearch. For example, signal data, instead of processing it before it isstored (resulting in loss of information), should be stored as it is andsignal processing techniques (e.g. FFT) then be used to extract arelevant view of the signal. This process is recursive in nature and, assuch, deciding the meaning of data (e.g., classifying, categorizing,segmenting, etc.) a priori cannot be performed.

Adding to this the rapidly widening gap between digital data productioncapabilities and network bandwidth capacity (at any scale), it becomesimperative to store the source data close to their point of productionand only distribute across the network the data relevant to the task athand.

Unfortunately, existing data management solutions (e.g., relationaldatabases, non-relational databases, data stores, and other datacollection techniques) have traditionally required static, pre-defineddatabase structures, rules and schema that are created for the databasewhen the database is established. As such, users requesting the data arelimited to data access according to static schema (that may beoutdated), from a database whose structure might be inefficient andcostly. Additionally, updating the database structure, rules or schemain existing solutions requires re-starting the database from scratch.

Others have put forth efforts towards adaptive database systems. Forexample, U.S. Pat. No. 5,983,218 to Syeda-Mahmood is directed tomodifying a relevance ranking of databases based query and responsepatterns for the databases. However, Syeda-Mahmood lacks any discussionof a modification of the databases themselves.

United States pre-grant publication number 2011/0282872 to Oksman, et al(“Oksman”) is directed to updating a system to increase theeffectiveness of future queries. However, in Oksman, the system'supdating is performed based on usage of query results or other feedbackto the query results, rather than based on the results themselves.Similarly, United States pre-grant publication number 2012/0296743 toVelipasaouglu, et al (“Velipasaouglu”) is directed to updating adatabase based on a query and a user's activity following a queryresponse.

United States pre-grant publication number 2007/0294266 to Chowdhary, etal (“Chowdhary”) is directed to using time-variant data schemas fordatabase management based on database modification requests. However, inChowdhary, the system simply stores new versions of schema stored alongwith older versions. Additionally, Chowdhary lacks any discussionregarding using query responses to generate new or updated versions ofdata schema.

All publications identified herein are incorporated by reference to thesame extent as if each individual publication or patent application werespecifically and individually indicated to be incorporated by reference.Where a definition or use of a term in an incorporated reference isinconsistent or contrary to the definition of that term provided herein,the definition of that term provided herein applies and the definitionof that term in the reference does not apply.

The following description includes information that may be useful inunderstanding the present invention. It is not an admission that any ofthe information provided herein is prior art or relevant to thepresently claimed invention, or that any publication specifically orimplicitly referenced is prior art.

In some embodiments, the numbers expressing quantities of ingredients,properties such as concentration, reaction conditions, and so forth,used to describe and claim certain embodiments of the invention are tobe understood as being modified in some instances by the term “about.”Accordingly, in some embodiments, the numerical parameters set forth inthe written description and attached claims are approximations that canvary depending upon the desired properties sought to be obtained by aparticular embodiment. In some embodiments, the numerical parametersshould be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof some embodiments of the invention are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspracticable. The numerical values presented in some embodiments of theinvention may contain certain errors necessarily resulting from thestandard deviation found in their respective testing measurements.

Unless the context dictates the contrary, all ranges set forth hereinshould be interpreted as being inclusive of their endpoints andopen-ended ranges should be interpreted to include only commerciallypractical values. Similarly, all lists of values should be considered asinclusive of intermediate values unless the context indicates thecontrary.

As used in the description herein and throughout the claims that follow,the meaning of “a,” “an,” and “the” includes plural reference unless thecontext clearly dictates otherwise. Also, as used in the descriptionherein, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise.

The recitation of ranges of values herein is merely intended to serve asa shorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g. “such as”) provided with respectto certain embodiments herein is intended merely to better illuminatethe invention and does not pose a limitation on the scope of theinvention otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element essential to thepractice of the invention.

Groupings of alternative elements or embodiments of the inventiondisclosed herein are not to be construed as limitations. Each groupmember can be referred to and claimed individually or in any combinationwith other members of the group or other elements found herein. One ormore members of a group can be included in, or deleted from, a group forreasons of convenience and/or patentability. When any such inclusion ordeletion occurs, the specification is herein deemed to contain the groupas modified thus fulfilling the written description of all Markushgroups used in the appended claims.

Thus, there is still a need for system that can dynamically adapt thestructure, schema and/or metadata of its data archives.

SUMMARY OF THE INVENTION

The inventive subject matter provides apparatus, systems and methods inwhich responses (“extracts”) to requests against a data store are usedto update a schema and/or structure of the database.

In some embodiments the data store is an archive of one or more sourcesof data. Archives might or might not be compressed, and might or mightnot include all of the data of the archived data sources, and might ormight not be have the same structure as the data source(s). The archivescan store data at full-fidelity (i.e., a reversible process, a bijectionbetween source data and stored data). Among other things, archives cancomprise one or more mirrors of the data source(s), collection(s) ofdata from the data source(s), as well as data from a sensor or othertransient data source that would not otherwise be stored. Archives aretypically considered to be write once-read many data stores, although itis contemplated that archives can grow by accruing data from additionaldata source(s).

Archives are preferably located logically proximate to their datasources, relative to end users, other archives or other intermediarynetwork components.

The schema includes metadata about the archive. Contemplated metadataincludes field names, data definitions, data types, access rules,traversal rules, strings used in executing historical extract requests,and statistical data regarding response data priority or other requestpatterns. Some or all of the metadata can advantageously be derived fromrequests, responses to the requests and/or processor, memory or otherperformance in executing the requests.

Embodiments can also include an analysis engine that performs thefunctions of updating the schema and/or structure of the data store.Contemplated updates include adding to, deleting and modifying the datadefinitions, data types or other metadata. Other contemplated updatesinclude compressing or re-arranging at least part of the data store.Some or all of the updates to the schema can advantageously be derivedfrom requests, responses to the requests and/or processor, memory orother performance in executing the requests.

Responses from the data store are preferably stored in a responserepository, and at least a portion of the response repository can bepublished on a network, for access by all manner of authorized entities,including for example requesting entities, and analysis engines notclosely associated with the data store.

Thus, the inventive subject matter can be used, for example, to providefull-fidelity storage of data while enabling end users to constantlyevolving ways to access and explore the data and retrieve what they needin a network-efficient manner.

Various objects, features, aspects and advantages of the inventivesubject matter will become more apparent from the following detaileddescription of preferred embodiments, along with the accompanyingdrawing figures in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a schematic overview of an exemplary system according to theinventive subject matter.

FIG. 2 is an overview of the information flow diagram within theexemplary system.

FIG. 3 is a flow diagram of the execution of processes and functions ofthe retrieval engine.

FIG. 4 is a flow diagram of the execution of processes and functions ofthe analysis engine.

DETAILED DESCRIPTION

Throughout the following discussion, numerous references will be maderegarding servers, services, interfaces, engines, modules, clients,peers, portals, platforms, or other systems formed from computingdevices. It should be appreciated that the use of such terms is deemedto represent one or more computing devices having at least one processor(e.g., ASIC, FPGA, DSP, x86-based CPU, ARM-based CPU, ColdFire-basedCPU, GPU, multi-core processors, etc.) configured to execute softwareinstructions stored on a computer readable tangible, non-transitorymedium (e.g., hard drive, solid state drive, RAM, flash, ROM, etc.). Forexample, a server can include one or more computers operating as a webserver, data store server, database server, or other type of computerserver in a manner to fulfill described roles, responsibilities, orfunctions. One should further appreciate the disclosed computer-basedalgorithms, processes, methods, or other types of instruction sets canbe embodied as a computer program product comprising a non-transitory,tangible computer readable media storing the instructions that cause aprocessor to execute the disclosed steps. The various servers, systems,databases, data stores or interfaces can exchange data usingstandardized protocols, interfaces and/or algorithms, possibly based onHTTP, HTTPS, AES, public-private key exchanges, web service APIs, knownfinancial transaction protocols, or other electronic informationexchanging methods. Examples of data exchange interfaces can includeEthernet, USB, HDMI, Bluetooth, wired, wireless, near-fieldcommunication interfaces, etc. Data exchanges can be conducted over apacket-switched network, the Internet, LAN, WAN, VPN, cellular, or othertype of packet switched network.

One should appreciate that the disclosed techniques provide manyadvantageous technical effects including enabling the constantrefinement of a data archive to decrease the computational cost ofexecuting data requests against the archive.

The following discussion provides many example embodiments of theinventive subject matter. Although each embodiment represents a singlecombination of inventive elements, the inventive subject matter isconsidered to include all possible combinations of the disclosedelements. Thus if one embodiment comprises elements A, B, and C, and asecond embodiment comprises elements B and D, then the inventive subjectmatter is also considered to include other remaining combinations of A,B, C, or D, even if not explicitly disclosed.

As used herein, and unless the context dictates otherwise, the term“coupled to” is intended to include both direct coupling (in which twoelements that are coupled to each other contact each other) and indirectcoupling (in which at least one additional element is located betweenthe two elements). Therefore, the terms “coupled to” and “coupled with”are used synonymously. Within the context of a networking environment,the terms “coupled to” and “coupled with” are also used euphemisticallyto mean “communicatively coupled with” where two or more network-enableddevices are able to exchange data over a network with each other,possibly via one or more intermediary devices.

FIG. 1 depicts a system 100 that generally includes a data component110, a processing and communication component 120 and a requestingcomponent 130.

The data component 100 generally includes at least one data source 111,at least one archive 112 at least initially derived from the at leastone data source 111, and at least one schema 113 associated with the atleast one archive 112. FIG. 1 illustrates an example whereby a dataarchive 112 is derived from a plurality of data sources 111, and has onecorresponding schema 113. However, it is contemplated that a dataarchive 112 can have a plurality of associated schema 113. Likewise, itis contemplated that that the data archive 112 can be derived from asingle data source 111. Conversely, in the simplest case, there is onedata source 111 that is archived at some point in time as a singlearchive 112, which is associated with a single schema 113.

From the perspective of the data archive 112, data source 111 can be anydevice from which the data used to create and update the contents of thedata archive 112 are received. Thus, generally speaking, data source(s)111 can include data storage devices (i.e. devices that store dataobtained from other sources), data creation devices (i.e., devices thatcan generate data but do not store it), and combination datastorage/generation devices (i.e., devices that can generate data, andstore generated and other data). Examples of data source 111 can includesensors (e.g. accelerometers, motion sensors, biometric sensors,temperature sensors, force sensors, impact sensors, flowmeters, GPSand/or other location sensors, pressure sensors, etc.), data storagedevices (e.g., server computers, non-transitory computer-computerreadable memory components such as hard drives, solid state drives,flash drives, optical storage devices, etc.), computing devices (e.g.,desktop computers, laptop computers, tablets, phablets, smartphones,etc.), and user-input devices (e.g., devices that receive data fromusers, which can include computing devices with user-input interfaces).

Data archive 112 can be considered to be a collection of data obtainedfrom data source(s) 111. The data archive 112 can be embodied via atleast one non-transitory computer readable storage medium that isconfigured to store the data of the data archive. In embodiments, thedata in the data archive 112 can be of the same data type as the data ofdata source(s) 111. In embodiments, the data archive 112 can importschemas, data definitions, and other data properties from the datasource(s) 111. In embodiments, the data in the data archive 112 can bein the same format as the source data from source(s) 111 (of the same ordifferent data types).

In embodiments, the data of archive 112 can be a full-fidelity versionof the corresponding data from source(s) 111. In these embodiments, abijection exists between the source data and the archive data, such thatthe source data in its original form can be reconstructed or regeneratedfrom the corresponding archive data.

In embodiments, the data archive 112 can comprise text data, whereby thedata from source(s) 111 is converted to text data for inclusion in dataarchive 112.

In embodiments, the data archive 112 can be a mirror of the datasource(s) 111.

In embodiments, data archive 112 can contain data in the form ofattributes and tuples specified in an input schema. The input schema canbe considered to be a default schema used in the creation of the archive112. Thus, the data archive 112 is the primary physical instantiation(data written to storage) of data in the system 100.

In these embodiments, the data archive 112 can be described as a tabulardata structure. Attributes are generally synonymous with columns,whereas records or rows are generally synonymous with tuples. A tuplecan represent one value for each attribute in an archive relative to anordered or fixed rank within each attribute. The rank may be relative toan ordering of some or all attributes and can be defined in the inputschema. Alternatively, collection of data within the archive 112 candefault to a rank based on structure and order of data received.

Generally speaking, the schema 113 can be considered to be the structureof the data archive 112, providing the organization of the data withinthe data archive 112. A schema 113 can include a definition of fields,data types, record delimiters, classes, relationship between data,compression rules, etc. The schema 113 can include performance metricthresholds for the execution of requests. The performance metricthresholds can be according to sections of data, particular records,request types, etc. The performance metric thresholds can includetargets for the execution of requests (e.g., time to completion,processor load, etc.), acceptable tolerances, etc. The thresholds can bedynamically adjusted based on factors such as network capability(overall and/or at a particular point in time), identity of requestor,the frequency of the data being accessed, etc. For example, for datathat is frequently accessed, the acceptable performance metricthresholds can be set to be more strict (i.e., only a slight drop inaccess speed is permissible).

As mentioned above, a data archive 112 can include a default (i.e.,input) schema 113 providing an initial structure of the abstract towhich initial requests to the archive 112 are applied to generate theinitial responses. For example, in an otherwise completely unstructuredand otherwise undefined data archive, a default schema can be thedesignation of record boundaries within the data archive which serve toprovide an initial structural organization to the data. These boundariescan be defined a priori by an administrator, or can correspond to knownproperties of data from source 111. This example is provided forillustrative purposes, and it is contemplated that default schema caninclude other and/or additional structural definitions, classifications,categorizations, etc. It is also contemplated that the data archive 112can lack any default schema whatsoever, wherein an initial schema can beconstructed via parsing and applying pattern recognition and rule-basedalgorithms to the data archive 112 by a processor (such as processor121).

In embodiments, the pattern recognition and rule-based algorithms can beapplied to the default schema, thus providing an initial step ofevolution to the basic default schema. In an illustrative example, anarchive can be created from sensor data, whereby the archive is createdbased on a simple read of the sensor data. In this example, the sensordata is loaded into an archive in tabular form where a row of values areassociated with a timestamp and sensor output (e.g., “Aug. 4, 201417:08:35, 75.9, 234.8, . . . ”). The initial archive schema can be theknown qualities of the sensor data, which in this example is the namesassociated with the field of the rows—“Date”, “Time”, “PumpTemperature”, “Pump Voltage”, etc. Upon applying pattern recognition onmultiple roles (preferably tens, hundreds, or even thousands or rows),aspects of the sensor data can be discovered to be periodic in nature.In this example, a voltage variation can be observed as having abaseline and a periodic multi-harmonic signal. Thus, a curve (e.g., awave) having a mathematical equation can be derived from the voltagedata. This default initial schema can be updated to incorporate thisdiscovery (e.g., via a set of parameters for a Fourier transformation(FFT)). Thus, a new set of operators can be applied to the data inprocessing user requests that were previously unknown. When users accessdata, they can request data associated with the harmonic signal part ofthe voltage data, whereby the processing component can use this updatedschema to perform the operations necessary to remove the baselinevoltage data from the generated responses, when the field is referencedin future requests. However, the underlying voltage data in the archiveis not modified or transformed at the physical/persistent level.

The schema 113 can include metadata associated with the archive 112,such as system metadata and archive metadata. System metadata containssystem configuration, performance, and consistency information that isboth created and updated. This metadata can be published and shared withother systems. For example, metadata structure shared with other systems(i.e., nodes in the larger ecosystem made up of a plurality of systemsof the inventive subject matter) can be updated to record the creationof a new archive through a messaging service. Archive metadata containsstatistical observations and indices calculated during storage engineoperations including compression and write. This metadata can be updatedalong with other aspects of schema 113.

The processing and communication component 120 generally includes atleast one processing component 121, memory 122, a retrieval engine 123,an analysis engine 124, and a communication component 125.

The processing component 121 can be one or more computer processors thatexecute instructions to carry out functions and processes associatedwith the inventive subject matter.

The retrieval engine 123 performs the functions associated withobtaining data from the data archive 112 in response to the extractrequest received from a requestor, and providing the data in the form ofa response back to the requestor, described in further detail herein.The analysis engine 124 performs the functions associated with modifyingthe schema 113 and/or the archive 112 based on the response, the schema113 and, in embodiments, the request. The functions and processesexecuted by analysis engine 124 are described in further detail herein.

In embodiments, the retrieval engine 123 and analysis engine 124 caneach comprise set of computer-executable instructions that are executedby processing component 121 to carry out their respective functions. Inthese embodiments, the retrieval engine 123 and analysis engine 124 canbe a single engine having the functions of both or be separate engines,and can be stored in either the same or separate non-transitorycomputer-readable media and can be executed by the same or differentprocessing component(s) 121.

Communication component 125 can include any communication interfaceenabling the processing component 120 to exchange data with the userinterface 132, via one or more data exchange networks, examples of whichinclude the data exchange interfaces, protocols and/or networksdiscussed herein.

The requesting component 130 generally includes at least one requestingentity 131 and at least one interface 132. The requesting entity 131 canbe considered to be the entity that initiates the request for data fromarchive 120, via the interface 320. The requesting entity 131 can be asingle user (as illustrated in the example of FIG. 1), a group of users,an organization, an enterprise, etc.

User interface 132 is an interface via which a requesting entity 131 cansubmit requests to access data contained in data archive 112. The userinterface 132 is presented to the user via a computing device, throughwhich the user can create the requests. The user interface 132 can be aweb-based interface hosted by an administrator of data archive 112,accessible via an internet browser on the computing device, anapplication executed on a requesting entity's 131 computing device, etc.As used herein, the term “user interface” can be considered to refer tothe software application as well as the computing device used to presentthe interface to the user and that enables the user to create requests.

As used herein, the term “requestor” is used to refer to the interface132 as the origin of the request, created according to the requestingentity 131. Thus, the term “requestor” may or may not include requestingentity 131 but always includes the interface 132.

As illustrated in FIG. 1, the groupings of system components into thedata component 110, the processing and communication component 120, andthe requesting component 130 are provided for illustrative purposesaccording to the various functions of the system components according toaspects of the inventive subject matter. Thus, the illustrated “groupingcomponents” 110,120,130 are not intended to limit or define thecontemplated physical embodiments of the system 100.

In embodiments, the data archive 112 is at relatively close dataproximity from one or more of the data source(s) 111, as compared to therequesting component 130. The term “data proximity” is intended to referto the relative difficulty in transmitting data from a sender to arecipient, which can be influenced by factors such as physicalproximity, size of data being transmitted, network capacity between thesender and receiver, number of intermediary nodes between the sender andreceiver, and other factors. Thus, in these embodiments, the dataarchive 112 is communicatively coupled to the data sources 111 in such away that the exchange of data from the data source(s) 111 to the dataarchive 112 is significantly faster than the exchange between dataarchive 112 and requesting component 130. This can be due to factorssuch as the data archive 112 being in closer physical proximity to thedata source(s) 111 than to requesting component 130, that the networkcapacity between the archive 112 and data sources 111 is greater thanthat of the network between archive 112 and requesting component 130(e.g., greater bandwidth, better-optimized network connection, etc.,less intermediary nodes slowing down data exchange, fewer bottlenecks,etc.), the size of information sent by individual data sources 111 toarchive 112 is smaller than the sending of all of the archive 112 to therequesting components, etc., or a combination of these factors.

The system 100 can also include an operator interface (not shown) thatallows an operator (e.g., a system administrator or other personnelhaving control over the data archive) to perform administrative andother service-related functions over the various aspects of the system.The operator interface can include one or more computing devicescommunicatively coupled to various components of the system 100. Anoperator can use operator interface to oversee the creation and loadingof data into a data archive 112, manage archive resources and computingenvironment, manage access control and security functions, etc. Forexample, an operator can trigger a manual alteration the nature of thefidelity of the data archive (potentially losing information). Thisalteration can be applied on an archive history basis, and isirreversible. However, the operator may elect to do so to conservestorage space when faced with storage constraint issues.

FIG. 2 provides an overview of the data flow processes of system 100,according to aspects of the inventive subject matter.

As shown in FIG. 2, the data archive 112 is initially created from datasource 111, illustrated via arrow 210. An extract request 220 to accessdata is generated by requestor 130 and transmitted to processingcomponent 120. Processing component 120 receives the extract request 220via communication component 125.

Retrieval engine 123 executes the received extract request 220 againstdata archive 112 according to the schema 113. Once the extract request220 has been executed, the retrieval engine 123 assembles the results inthe form of response 230 (also referred to as extract response 230).Once the response 230 has been generated, it is transmitted back to therequestor 130. FIG. 3 is a flow chart of the processes executed byretrieval engine 123 in greater detail.

Once the response 230 has been generated, analysis engine 124 analyzesthe response 230 and performs an update 240 to at least one of (a) theschema 130 associated with the archive 112 and (b) the structure of thedata archive 112 itself based on the response 230. FIG. 4 illustratesdata flow processes associated with an analysis engine 124 in greaterdetail.

As shown in FIG. 3, retrieval engine 123 receives the request 220 atstep 310. At step 320, the retrieval engine 123 applies the request 220to the schema 113 to determine the extent to which the informationsought in the request 220 is defined by fields or other structuralorganizational scheme within archive 112. In embodiments, the retrievalengine 123 can include data access control functions whereby credentialsof requestor 130 are verified prior to allowing any access to thearchive 112. The access control functions can include verification ofthe identity of the requesting entity 131, verification a networkaddress, authentication procedures (e.g., passwords, encryption schemes,certificates from an authority, etc.), role-basedauthentication/verification (e.g., a role within an organization, etc.),and other forms of access control.

In embodiments, the request 220 can be formatted to include all of thefields (or other structural categorization) sought in the data request.For example, the request 220 can include one or more extract requestparameters in the format “field name=field value”. Thus, the retrievalengine 123 matches each of the field names in the extract requestparameters with the field names defined by the schema 113 for thearchive 112. Other extract request parameters can include data type,data size, length, etc. Extract request parameters can also becombinations of single parameters. For example, in the “field name=fieldvalue” example, the request 220 can also specify that the “field value”be of a certain data type (e.g. string, integer, etc.), have a certainmaximum or minimum length, etc.

In embodiments, the request 220 can be formatted according to a naturallanguage question, in which case the processing component 120 caninclude a semantic database to determine that the request 220 isfocusing on a particular set of fields, and then compare the fieldsderived from the request 220 against the defined fields according toschema 130.

At step 330 the retrieval engine 123 executes the operations associatedwith carrying out the request 220 according to the field values of thedefined fields of the schema 113, such as filtering the records of thearchive 112 such that the output of the operation is those recordsmatching the field values of the defined fields.

It should be noted that a request 220 can include a request for datawhose field (in this example), data type, data format, data definitionor other organizational/structural parameter is not defined or otherwiseknown in the data archive 112 according to the schema 130. For theseunknown fields, the matching performed at step 320 with known/definedfields as set forth in the schema 113 will fail to produce a match, andcan be flagged or otherwise identified by the retrieval engine 123 asunknown fields. At step 340, the retrieval engine 123 executes amatching of the “field value” of extract request parameter with thearchive 112 to determine whether the field value corresponds to any partof any record within the archive 112. The match can be a literal (i.e.,exact) match or can be a proximity match (i.e., matching within adefined percentage of similarity).

In embodiments, any matches can be analyzed to infer othercharacteristics of the matched data. For example, for every match of afield value, the retrieval engine 123 can determine the data type of thefield value (e.g., that the match is a string, integer, etc.).

At step 350, the retrieval engine 123 performs the operations on thearchive 112 (e.g., filtered or otherwise processed) according to thesematches to return the data output used in generating the response 230.

As illustrated in FIG. 3, step 340 is executed after the processes ofstep 330 are executed. Thus, the matching of the field value of theextract request parameter of the unknown field is limited to thoserecords returned from the filter processes performed with the definedfields of archive 112. However, in embodiments, step 340 can be executedprior to step 330, whereby the matching of the values of unknown fieldscan be performed against all of the data within the archive 112.Therefore, steps 330 and 350 of these embodiments are effectivelycombined.

At step 360, the response 230 is generated based on the output of theexecution of the extract request, and provided back to requestor 113.

The response 230 can be considered to be a view of the archive 112presented to the requesting entity 131 via the user interface 132. Inembodiments, the response 230 can be a set of scalar expressions (e.g.,scaling and compare expressions, etc.) that define the set of data inthe archive (or a projection/subset thereof) that correspond to the datarequested by the requestor 130.

In embodiments, the set of scalar expressions can include clauses thatdescribe the Projection, Function and Filter type. A Projection can beconsidered as a selection of a sub-set of data (such as a subset ofcolumns of all available columns, if the archive is so structured). TheProjection can also be used to modify an attribute's value. The Functioncan be a modification of attributes within the projection. Examples caninclude scalar mathematical functions such as addition and subtraction.The Filter can conditionally restrict tuples within the definedProjection. In embodiments, the Function can be applied to either aFilter or a Projection.

The following are examples of a request 220 executed on data archives112 having different levels of schema definition, according to theprocess described in FIG. 3. The request 220 in this example is seekingdata associated with males aged 35-40 years old, living in the city ofOrange, Calif. Thus, the request 220 is considered to include thefollowing “field=value” parameters: ‘gender’=‘male’, ‘age’=‘35-40’, and‘city’=‘Orange’.

EXAMPLE 1

The extract request 220 is applied to a data archive 112 havingestablished, known data fields for all of the extract request parameters(‘gender’=‘male’, ‘age’=‘35-40’,and ‘city’=‘Orange’) of the request 220.For each record in the archive 112, there are defined fieldscorresponding to “gender”, “age” and “city”. Correspondingly, theextract request 220 is formatted according to these known fields ofarchive 112. Thus, the retrieval engine 123 executes the extract request220 and filters the data in the archive 112 according to the gender,age, and city fields. In this example, there are no “unknown” fields inthe request 220, so steps 340 and 350 of FIG. 3 are not executed. Thisresult is then used to generate response 230 at step 360.

EXAMPLE 2

The same request 220 from Example 1 is applied to a “less established”archive 112 (i.e., the schema 113 is less established), where some, butnot all, of the fields corresponding to the parameters in the request220 are known/defined in the archive 112. In this example, the schema113 includes defined “gender” and “age” fields, but does not have adefined “city” field for archive 112. Having determined the definedfields at step 302 and executed the functions according to those definedfields at step 330, the retrieval engine 123 executes step 340 andsearches within the results of step 330 for the literal match “Orange”(in this example, a literal match is preferred because the city namewill not have a plural or other conjugation). Once the matches areobtained, the processes of step 350 are executed and the responsegenerated at step 360. In the embodiments whereby step 340 is executedprior to step 330 as described above, the retrieval engine 123 performsmatch of “Orange” against the entire archive 112.

EXAMPLE 3

In this example, the request 220 is applied to an even less“established” archive 112 (i.e., having an even less established schema113), where none of the field names are known, such that none of thefields contained in the request 220 will match with corresponding fieldsof archive 112. In this example, the schema 113 can include otherdefined fields (but none that match the request 220's fields), or canhave no defined fields of any kind. Thus, the only “knowns” are therecord boundaries defined by the schema 130. In this case, the executionof step 320 will not return any defined fields. Thus, the retrievalengine 123 executes the matching of step 340 for literal match for“male”, a literal integer match of “35-40” (and can include matches ofeach integer 35, 36, 37, 38, 39, 40), and the literal match of “Orange”.

EXAMPLE #4

This example is similar to Example 3, but the record boundaries are alsonot “known”. As defined herein, a record boundary indicates a beginningand an end of each record (e.g., a row in a spreadsheet, etc.). In otherwords, the data archives only has one dimension (e.g., flat) including along single string of data. In this example, the retrieval engine 123searches the entire archive 112 for matches of the field values in therequest 220. Based on the matches, the retrieval engine 123 can inferrecord boundaries accordingly by performing pattern analysis on thematched results (e.g., periodicity of repeating matches, and thedistance between the repeating matches, taking into account that not allfield values will match in all records, etc.).

Having inferred the record boundaries, an offset can be determined foreach match from what are inferred to be separate records, to account forpossible different field value lengths among a same field type, class ofliterals (e.g., male, male, female, male), or data type (e.g., integer,floating point, etc.). For example, “male” may return matches that infact are for “female”. However, because a match of “male” within“female” will have an offset of two corresponding to the “f” and “e”characters, these results can be eliminated as false positives for thepurposes of generating the response 230.

In embodiments, the record boundaries inference can be executed by theretrieval engine 123 even if record boundaries/delimiters exist withinarchive 112, such that corrections and adjustments to previouslydetermined record boundaries can be performed over time as the archive112 is accessed by additional requesters and additional responses areprovided thereto.

FIG. 4 illustrates data flow processes associated with an analysisengine 124, whereby the analysis engine 124 uses the response 230 tomodify the schema 113 and/or the structure of the archive 112 itself.The functions and processes of FIG. 4 can be considered collectively tobe the functions and processes available to the system to perform one ormore updates 240 of FIG. 2.

At step 410, the analysis engine 124 accesses the generated response 230and any additional corresponding data (if not included within theresponse 230 itself). The use of “accessing” the generated response 230is intended to refer generally to the logical step that the generatedresponse 230 becomes available to the analysis engine 124 for thepurposes of carrying out its associated functions. As such, the analysisengine 124 can also be considered to be “receiving” the generatedresponse 230. For example, the generated response 230 can be accessed bythe analysis engine 124 prior to transmission as a logical stepfollowing step 360, constituting a logical “hand-off” of the response230 from the retrieval engine 123 to the analysis engine 124. In anotherexample, a copy of the generated response 230 is generated by theretrieval engine 123 and provided to the analysis engine 124 such thatthe functions of the analysis engine 124 can be performedchronologically independent of the actual transmission of the response230. In another example, in embodiments where the retrieval engine 123and analysis engine 124 are part of a single engine, “accessing” canrefer to the invocation of the functions associated within the analysisengine 124 and applied to the response 230 (or a copy thereof).

The response 230 generated at step 360 of FIG. 3 can include performancemetrics associated with the retrieval of the requested data and thegeneration of the response 230. Performance metrics can include a timeto complete the request, a resource load indication (e.g., processorusage, energy usage, memory usage, etc.), and other performance metrics.In embodiments, the performance metrics are a part of the response 230that is provided to the requestor 130. In embodiments, the performancemetrics are generated at step 360 along with, but are separate from, theresponse 230. In these embodiments, the performance metrics can beprovided to the analysis engine 124 along with the response 230 withoutalso having to provide the performance metrics to the requestor 130.

At step 420, the analysis engine 124 modifies the archive 112 and schema113 according to any new archive structure component (e.g., new record,field, data type, delimiter, etc.) reflected in the response 230. Step420 can be executed according to steps 421-422.

At step 421, the analysis engine 124 applies any new record delimitersto the schema 113 based on the location of the record delimitersinferred by the retrieval engine 123 (such as in Example 4 above).

The modifications to schema 113 can include data the location datawithin the archive 112 of the created record delimiters. Additionalupdates to the schema 113 can include a determination on the sizes ofnewly established records. Modifications to the archive 112 itself caninclude insertion of record delimiters or other record boundaries at thecorresponding locations according to the updated schema 113, as well asmodifying the newly established records for consistency with a desiredrecord structure (e.g. inserting or removing spaces, lines, etc. toorganize the records within the archive 112). If record delimitersalready exist in archive 112, or do not require adjustment, the analysisengine 124 can skip step 421.

At step 422, the analysis engine 124 applies any new fields, data types,data definitions, or other intra-record structuralparameters/definitions to the schema 113. This can include thedetermined locations within corresponding records and/or the overallarchive 112 and any correlations with other structural parameters (e.g.,a particular field name also has values of a particular data type). Aswith the record delimiters, intra-record structural delimiters cansimilarly be applied to the archive 112 itself.

Thus, in Example 2 discussed above, the schema 113 is updated to includea “city” field name, at the appropriate locations within the archive112, which can be an established offset from the record boundaries foreach record. This then aligns data in records as corresponding to a“city” field, even in records that did not contain the “Orange” match.Additionally, if the schema 113 has been updated to include that acorrelation of a “city” field as having values of a “string” or “text”type, the analysis engine 124 can analyze the non-matched records toverify that the same data type exists in those records, and the size ofthe records. The analysis engine 124 can then update rules associatedwith the expected (or permitted, maximum, or minimums, etc.) size offield values for cities within the schema 113.

In embodiments, the modifications to the archive 112 can be performed bythe analysis engine 124 as soon as the modifications to the schema 113are performed. In other embodiments, the modifications to the archive112 are applied only when a subsequent request 220 from a requester 130(either the same requestor or a different one) is executed. Thus, thearchive 112 itself is only modified with the delimiters and othermodifications at run-time.

In addition to the schema modifications discussed above, modificationsto the schema 113 can include updating the schema 113 to reflectobserved characteristics of the archive 112 as a whole. Examples ofthese characteristics can include recognitions of periodicity, decay,etc., such as the example of the sensor data illustrated above.

Preferably, requestors can only perform extraction actions against thearchive (i.e., requests for access to data and receive responses). Whilethe archive and/or the schema can be modified based on the responseprovided to the requestor, a requestor cannot directly modify thearchive data itself. However, to allow requestors to narrow or filterdata presented via a response, requestors can mark (e.g. viaannotations, flags, etc.) data at any granularity and maintain thosemarkings for any length of time. The markings can be used by the systemto keep track of data not deemed relevant in the temporal semantic viewof the archive as seen by the requestor. In embodiments, the markingscan be reintroduced into the system by interpreting them as newrequests, which can be constructed onto the extracted response or as anew request combining the prior request and the “marking” request.

In embodiments, certain modifications to the archive 112 can be labeledas “provisional” modifications within schema 113, such that they areconsidered preliminary or subject to further modification. Provisionalmodifications can include modifications that have been recently created(e.g., within a certain number of requests processed against the archive112). After provisional modifications have withstood a pre-determinednumber of requests and responses processed against the archive 112,those modifications can be made permanent (i.e., serving to confirm thatthe fields, data definitions, types, record boundaries, etc. are valid).

At step 430, the analysis engine 124 analyzes performance metrics ofresponse 230 against the performance metrics thresholds of the schema113. If the performance metrics of the response 230 exceed or otherwisefall outside of desirable or acceptable thresholds as indicated byschema 113, the analysis engine 124 can execute changes to the dataarchive 112 to reduce the computational cost or load to execute requestson similar data in the future at step 431. For example, the analysisengine 124 can move records within the archive 112 such that the recordsare in a location within the archive 112 that is more quickly accessedduring the execution of a retrieval process. In another example, theanalysis engine 124 can modify the schema 113 such that the filter orderamong several fields is optimized.

In embodiments, the request 220 can be re-executed by the retrievalengine 123 (at step 432) after the changes of step 431 are implemented,and the new performance metrics compared against the performance metricsof the response 230 to verify that the performance of executing theextract request has improved.

At step 440, the analysis engine 124 can compress sections of datawithin the archive 112. For example, sections of data that remainunknown (either within records our between records) can be compressed.In another example, defined sections of data (of a particular fieldtype, data type, data definition, etc.) can be compressed to takeadvantage of commonalities and redundancies.

In FIG. 4, the processes 420-422, 430-432 and 440 executed by analysisengine 124 are shown in parallel to show that they can be executedconcurrently by the analysis engine 124. However, it is contemplatedthat the processes can be executed in series in the numerical order ofthe elements described in FIG. 4, or in other sequential orders.Additionally, the extent to which the archive 112 and/or the schema 113can be modified performed via the processes shown in FIG. 4 can begoverned by priority rules. For example, a reorganization of a recordwithin the archive 112 such as in step 431 may be limited or outrightrejected if superseded by a higher-priority rule (such as the movementof the record pushes other records “down” that are frequently requestedby users and thus, must be maintained at the “top” of the accessibilitylist).

It is contemplated that the modifications to the archive 112 and schema113 described in FIG. 4 can also be enacted based on a collection ofhistorical responses generated in response to historical extractrequests submitted by one or more requestors in the past. Additionally,the response 230 can be added to the collection of historical responses.Metadata of schema 113 can statistical data and analysis includingrequest patterns (e.g. from which requesters, how frequently, which datahas been accessed, etc.). As such, the analysis engine 124 can establishand update data access priorities for certain sections of data withinthe schema 113. Historical responses, requests, and other data can bestored in a response repository, which can be memory 122 of component120, the same storage as the archive 112, or another, separatenon-transitory computer-readable medium.

In embodiments, the analysis engine 124 can also modify the archive 112and/or the schema 113 based in part on the received request 220.Similarly, where historical responses are used to shape the archive 112and schema 113, so can historical requests be used.

In embodiments, system 100 will be a part of a larger ecosystem havingother, similar systems with corresponding archives generated based oncorresponding data sources. In these embodiments, the system 100 canalso include a publication module that can publish the existence of thearchive 112 to other systems in the ecosystem. It is furthercontemplated that the schema 113, metadata within the schema 113, andother characteristics of the archive 112 can be published. Similarly,the response repository (which may or may not include a collection ofrequests), can be published via the publication module.

In embodiments, the response 230 can be a data stream. In theseembodiments, the steps of FIG. 4 can be applied as the data stream isoccurring, based on data being transferred as needed during the stream.For example, the analysis engine 124 can modify the archive 112 and/orschema 113 based on the response cost during the data stream. For datastreams, the modifications to the archive 112 and/or schema 113 can betransitory for the duration of the stream so as to provide the immediatebenefit of the modifications. In embodiments, these modifications canalso be made permanent as described herein.

It should be apparent to those skilled in the art that many moremodifications besides those already described are possible withoutdeparting from the inventive concepts herein. The inventive subjectmatter, therefore, is not to be restricted except in the spirit of theappended claims. Moreover, in interpreting both the specification andthe claims, all terms should be interpreted in the broadest possiblemanner consistent with the context. In particular, the terms “comprises”and “comprising” should be interpreted as referring to elements,components, or steps in a non-exclusive manner, indicating that thereferenced elements, components, or steps may be present, or utilized,or combined with other elements, components, or steps that are notexpressly referenced. Where the specification claims refers to at leastone of something selected from the group consisting of A, B, C . . . andN, the text should be interpreted as requiring only one element from thegroup, not A plus N, or B plus N, etc.

1. (canceled)
 2. A method comprising: obtaining a set of search resultsgenerated in response to a first search query, wherein the set of searchresults identifies a subset of the plurality of records having attributefields defined by an initial schema; determining whether the identifiedsubset of the plurality of records includes data that is associated witha predetermined relationship; and in response to determining that theidentified subset of the plurality of records includes data that isassociated with the predetermined relationship, generating a modifiedschema based on the predetermined relationship, wherein the modifiedschema includes a different set of attribute fields than the first setof attribute fields.
 3. The method of claim 2, wherein generating themodified schema based on the predetermined relationship includes addingan additional attribute field to the initial schema.
 4. The method ofclaim 3, the method further comprising: for each particular record ofthe subset of the plurality of records: deriving a data value based ondata from the particular record that is associated with at least oneattribute field of the initial schema; and associating the derived datavalue with the additional attribute field.
 5. The method of claim 4, themethod further comprising: receiving a second search query from arequesting entity after generation of the modified schema, wherein thesecond search query requests the derived data value that is associatedwith the additional attribute field; and providing the derived datavalue that is associated with the additional attribute field to therequesting entity.
 6. The method of claim 5, wherein the requestingentity includes a device associated with a single user.
 7. The method ofclaim 2, wherein determining whether the identified subset of theplurality of records includes data that is associated with apredetermined relationship comprises: applying one or more rules to eachrecord in the identified subset of the plurality of records.
 8. Themethod of claim 2, further comprising: determining that the identifiedsubset of the plurality of records includes data that is associated witha predetermined relationship if the subset of the plurality of recordsincludes data that has one or more periodic characteristics.
 9. A systemcomprising one or more computers and one or more storage devices storinginstructions that when executed by the one or more computers cause theone or more computers to perform operations comprising: obtaining a setof search results generated in response to a first search query, whereinthe set of search results identifies a subset of the plurality ofrecords having attribute fields defined by an initial schema;determining whether the identified subset of the plurality of recordsinclude data that is associated with a predetermined relationship; andin response to determining that the identified subset of the pluralityof records includes data that is associated with the predeterminedrelationship, generating a modified schema based on the predeterminedrelationship, wherein the modified schema includes a different set ofattribute fields than the first set of attribute fields.
 10. The systemof claim 9, wherein generating the modified schema based on thepredetermined relationship includes adding an additional attribute fieldto the initial schema.
 11. The system of claim 10, the operationsfurther comprising: for each particular record of the subset of theplurality of records: deriving a data value based on data from theparticular record that is associated with at least one attribute fieldof the initial schema; and associating the derived data value with theadditional attribute field.
 12. The system of claim 11, the operationsfurther comprising: receiving a second search query from a requestingentity after generation of the modified schema, wherein the secondsearch query requests the derived data value that is associated with theadditional attribute field; and providing the derived data value that isassociated with the additional attribute field to the requesting entity.13. The system of claim 9, wherein determining whether the identifiedsubset of the plurality of records includes data that is associated witha predetermined relationship comprises: applying one or more rules toeach record in the identified subset of the plurality of records. 14.The system of claim 9, the operations further comprising: determiningthat the identified subset of the plurality of records includes datathat is associated with a predetermined relationship if the subset ofthe plurality of records includes data that has one or more periodiccharacteristics.
 15. A computer program product encoded on one or morenon-transitory computer readable media, the computer program productcomprising instructions that when executed by one or more computerscause the one or more computers to perform operations comprising:obtaining a set of search results generated in response to a firstsearch query, wherein the set of search results identifies a subset ofthe plurality of records having attribute fields defined by an initialschema; determining whether the identified subset of the plurality ofrecords include data that is associated with a predeterminedrelationship; and in response to determining that the identified subsetof the plurality of records includes data that is associated with thepredetermined relationship, generating a modified schema based on thepredetermined relationship, wherein the modified schema includes adifferent set of attribute fields than the first set of attributefields.
 16. The computer program product of claim 15, wherein generatingthe modified schema based on the predetermined relationship includesadding an additional attribute field to the initial schema.
 17. Thecomputer program product of claim 16, the operations further comprising:for each particular record of the subset of the plurality of records:deriving a data value based on data from the particular record that isassociated with at least one attribute field of the initial schema; andassociating the derived data value with the additional attribute field.18. The computer program product of claim 17, the operations furthercomprising: receiving a second search query from a requesting entityafter generation of the modified schema, wherein the second search queryrequests the derived data value that is associated with the additionalattribute field; and providing the derived data value that is associatedwith the additional attribute field to the requesting entity.
 19. Thecomputer program product of claim 18, wherein the requesting entityincludes a device associated with a single user.
 20. The computerprogram product of claim 15, wherein determining whether the identifiedsubset of the plurality of records includes data that is associated witha predetermined relationship comprises: applying one or more rules toeach record in the identified subset of the plurality of records. 21.The computer program product of claim 15, the operations furthercomprising: determining that the identified subset of the plurality ofrecords includes data that is associated with a predeterminedrelationship if the subset of the plurality of records includes datathat has one or more periodic characteristics.